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1.  TCP  Implementation 

At  the  date  of  this  writing,  the  Transmission  Control  Program  (TCP) 
at  Stanford  University  -■  Digital  Systems  Laboratory  (SU-DSL)  was  complete, 
except  for 

(a)  code  to  perform  desynchronization 

(b)  code  to  handle  the  arrival  of  FIN  messages  when  the  TCP  is  in  a 
state  other  than  ESTABLISHED.  A proposal  has  been  circulated  to 
the  other  participants  in  the  internetwork  experiments  but  comments 
have  not  yet  been  received. 

1.1  New  CLOSE  Procedure 

During  tfie  period,  a change  was  made  to  the  connection  closing 
mechanism  which  required  reprogramming  of  the  CLOSE  code  as  well  as  the 
TCP  (Transmission  Control  Block)  deletion  code,  since  TCB's  can  now  only  be 
deleted  when  the  user  has  CLOSEd  the  connection  and  FIN's  have  been  exchanged 
AND  acknowledged.  The  latter  feature  is  the  new  procedure  and  guarantees 
that  the  initiator  of  the  CLOSE  will  be  able  to  distinguish  between  the  case 
that  the  FIN  was  received  normally  at  the  destination  and  the  case  that  the 
connection  has  somehow  been  terminated  abnormally  at  the  remote  side.  In  the 
earlier  design,  it  v^as  sufficient  to  send  and  receive  a FIN  and  to  get  a local 
CLOSE  from  the  user.  However,  a FIN  send  in  response  to  a FIN  received  might 
not  in  fact  arrive,  and  the  retransmission  of  the  originating  FIN  would  then 
get  a "connection  non-existent"  message  in  I'eturn,  leaving  tfie  initiator  of 
the  close  uncertain  whetfK'r  fiis  FIN  htid  arrivc’d  or  not. 


] .?  ICF  si/e  arid  org.ini z.i tion 

The  SU-DSL  TCP  is  written  to  run  under  nrj  ELF  operating  system  kernel. 

The  etjui[>:nent  at  SU-DSL  is  shown  in  figure  1.  Ihe  basic  software  organization 
is  shown  in  figure  2 and  the  approximate  sizes  of  the  various  TCP  software 
components  sh.own  in  Table  1.  Initinlly,  most  of  the  TCP  was  programmed  in 
fiCPL  (Basic  Compatible  Programming  Language). 

Each  module  in  Figure  2 is  a separate  ELF  process  running  independently. 
The  user  calls  are  serviced  by  the  user  call  interface  code,  reached  by 
Emulator  Traps  (EMT's).  Tiie  other  processes  communicate  among  themselves  by 
sending  and  receiving  signals. 

As  can  be  seen  in  Table  1,  tfje  size  of  the  TCP  leaves  little  room  in  the 
28k  word  memory  of  the  PDj^-11/20  for  experimental  user  software.  A part  of 
the  problem  is  the  cost  of  the  BCPL  fiigh  level  language,  lo  remedy  some  of 
the  space  difficulties,  v/e  plan  to  reprogram  parts  of  the  TCP  in  MACNll 
assembly  language.  The  generality  of  the  implementation  (arbitrary  number 
of  connections,  dynamic  buffer  allocation,  etc.)  has  an  undeniable  space 
cost  (see  section  1.3  below  on  Packet  Radio  TCP-0  implementation).  The  lowest 
level  service  routines,  heretofore  written  in  ELF  are  being  reprogrammed  in 
assembly  language  with  space  reductions  up  to  70/^  in  some  cases. 

1.3  Single  Connection  TCP  for  Packet  Radio  Terminal 

We  have  ordered  an  LSI-11/03  with  8k  words  of  memory,  a 16  bit  parallel 
interface  and  a terminal  interface.  We  plan  to  build  an  interface  which 
conforms  to  BBN-1822  for  the  purpose  of  attaching  the  terminal  to  a Packet 
Radio  unit  (PRO).  Initially,  we  will  use  an  LSl-11/03  at  SRI  to  test  our 
TCP/TELNET  code,  until  we  have  a PRO  of  our  own  delivered  to  SU-DSL.  The 
PRU  will  be  used  both  to  connect  the  PDP-11/03  terminal  to  the  PRNLT  (Packet 
Radio  Net)  and,  alternatively  to  connect  our  PDP-11/20  as  a host.  Lventually, 
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wo  hope  to  test  tcrmiruil  , host,  ond  station  concurrently.  Figure  3 
illustrates  the  terminal  hardware  and  figure  4 (and  table  2)  the  software 
plan. 

We  are  hoping  to  fit  tfie  entire  terminal  software  package  in  4k  words, 
but  are  not  sure  how  much  code  the  host/PRNET  interface  will  require  in 
the  way  of  support  software.  In  atiy  case,  the  TCP,  which  only  runs  a 
single,  full-duplex  connection,  should  not  require  more  than  2.5k  words, 
handcoded  in  assembly  language  (MACNll). 

A preliminary  software  specification  for  TCP-0  is  to  be  delivered  by  15 
November  and  a final  specification  with  implementation  and  documi.'ntation 
complete  by  1 February  1976. 

1.4  PRNET  Host  softv/are/hardware 

In  addition  to  the  PRNET  terminal,  we  have  installed  an  IMP-llA 
interface  for  our  PDP-11/20  and  are  awaiting  delivery  and  installation  of  the 
PRO.  Initially,  this  unit  will  be  installed  in  the  SU-DSL  machine  room 

with  its  antenna  on  the  roof.  We  have  initiated  work  to  pull  cables  to 
the  roof  of  the  ^ story  Durand  building  nearby  and  plan  to  attach  the  PRO 
antenna  to  one  of  the  existing  Durand  microwave  towers.  Sufficient  rack 
space  for  the  PR  repeater  has  been  allocated  in  the  Instructional  TV  Facility 
equipment  room  and  an  11-15  pair  cable  will  be  run  from  the  roof  of  Durand 
to  its  basement,  as  shown  in  figure  5.  The  high  antenna  should  give  excellent 
range  to  the  PRO  (e.g.  to  SRI  and  Eichlerville  units).  Installation  of  the 
PRO  is  expected  in  the  middle  of  February  1976  (horseback  estimate). 

Software  for  the  PDP-11/20  PRNET  host  will  include  ELF,  a reduced  TCP, 
PRNET/HOST  software,  IMP-llA  driver,  and  various  simple  service  routines 
(e.g.  server  TELNET  and  some  as  yet  unspecified  interactive  application 
programs).  If  the  general  TCP  proves  too  large,  we  can  demonstrate  the  host 
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usincj  TCP-0,  lr'[  Uiib  would  require  the  cudiriq  of  a server  TELNLT  to 
interface  to  TCP^U.  We  hope  this  won't  be  fiecessary. 

2.  Experiments 

2.1  Internet  packet  exchaiujes 

Connections  fiave  been  establishied  between  SU-DSL/BBN  and  SU-DSL/UCL. 

Data  has  been  excfianged.  SU'DSL  has  always  been  the  initiator  since  other 

sites  do  not  yet  have  an  exerciser  (terminal  controller  - althougfi 

BBN  may  have  a primitive  TELNET  nearly  completed).  All  sites  (BBM,  UCL, 

SU‘DSL)  have  opened  looped  connections  to  themselves,  but  only  BBN  and 
SU-DSL  have  successfully  CLOSED  connections  (UCL  is  writing  CLOSE  code 
and  FIN  handl i ng  now) . 

BBN  has  a functioning  echoer  and  SU-DSL  has  opened  connections  to  it, 
sent  data,  and  closed  the  connectiofi  (a  syntax  error  in  SU-DSL  code  caused 
its  TCP  to  crash  before  the  TCB  was  actually  removed,  but  this  has  been 
repaired) . 

2.2  Initial  Performance  Tests 

So  far  as  we  know,  the  BBN  PDP-10  implementation  is  still  using 
dSYS  traps  and  therefore  runs  at  about  1 letter/second.  We  have  been 
testing  throughput  by  opening  a connection  to  ourselves  (send  and  receive 
ports  are  looped).  As  a result,  we  have  begun  to  explore  changes  in  im- 
plenientation  choices  to  alleviate  some  bottlenecks. 

One  of  tfic  goals  of  the  TCP  implementation  was  to  allow  for  the  "piggy- 
backing" of  acknowledgments  on  return  traffic.  In  the  absence  of  such  traffic, 
of  course,  empty  packets  with  acknowledgment  information  must  be  forced  out 
to  avoid  unnecessary  retransmissions  from  tlie  sender.  In  our  initial  im- 
plementation, new  arriving  packets  cause  a "NEWRECEIVE"  flag  to  be  set. 
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If  packets  ore  produced  and  sent  out  the  reverse  channel,  tiie  accompanying 
acknowledgnient  causes  this  flag  to  be  reset.  The  retransmission  process 
maintains  a queue  of  wake-up  times  (one  per  connection)  arranged  in  order, 
nearest  event  first.  When  it  awakens,  it  checks  to  see  if  a "HEWRECEIVC"  flag 
is  set  for  this  connection  and  sends  out  an  empty  acknowledgment  before 
resetting  the  flag  and  goes  on  to  check  whether  any  packets  still  on  the 
retransmission  queue  should  be  retransmitted.  We  chose  not  to  modify  the 
acknowledgment  information  in  retransmitted  packets  since  this  would  require 
modification  of  the  packet  checksum  and  might  lead  to  serious  confusion  at 
the  destination  TCP  when  dealing  with  Gateway  fragmentation.  After  servicing 
a connection,  the  retransmission  routine  determines  the  earliest  time  at  which 
it  should  next  awaken  and  schedules  a signal  for  this  time.  If  the  earliest 
required  time  is  greater  than  a default  constant  for  the  retransmission  process 
(e.g.  k - h second),  then  the  process  v^akes  itself  after  the  shorter  interval. 

This  arrangement  can  lead  to  some  odd  interactions  when  sending  the  output 
of  a full  duplex  connection  directly  into  the  inpui  side,  as  shown  in  figure  6. 

In  the  self  loop  experiment,  a connection  is  established  between  the 
send  and  receive  sides  of  the  same  port.  Letters  are  sent  and  received  and 
various  statistics  collected  to  highlight  the  behavior  of  the  TCP. 

2.2.1  Single  letter  at  a time  throughput 

A user  program  was  written  (modification  of  the  conventional  exerciser) 
which  would  send  a letter  of  a fixed  length,  and  wait  until  it  was  acknowledged 
at  which  time  a second  letter  was  sent.  This  corresponds  roughly  to  RFK’M 
(request  for  next  message)  driven  NCP  experiments. 

Letter  length  (i.e.  actual  text  length  in  octets)  was  varied  as  shown  in 
Table  3.  The  packet  retransmission  timeout  v/as  synonymous  with  the  retransmission 
process'  default  wake-up  timeout. 


rhi’  result/',  of  Uiis  exper iment  clearly  show  that  when  no  rev(:rse 
traffic  is  waiting  (i.e.  in  this  case,  only  one  message  at  a time  is  being 
sent),  then  all  acknowledgments  ore  sent  by  the  retransini ssion  process. 

Furt itermore , the  number  of  unnecessary  retratismissions  increases  as  the 
retransiiti ss iori  process  wake  up  time  decreases  (,ond  packet  retransmission  time 
decreases  correspondingly).  In  fact,  for  the  0.?.b  second  retransmission 
timeout,  there  were  sometimes  more  retransmissions  than  letters  sent.  Two 
problems  were  evident,  first,  the  retransmission  process  wake-up  time  was 
unnecessarily  tied  to  the  packet  retransmission  time  and  second,  ACKs  were 
not  getting  to  the  source  fast  enough.  Of  course,  in  this  self  loop,  there 
may  be  some  intei'action  among  the  send/rccei ve  sides  of  the  TCP  operating  on 
the  same  port  (e.g.  locks  for  TCB  state  information,  priority  of  TCP  and 
user  process,  etc.)  which  would  riot  ordinarily  be  found  in  a connection  to 
a port  at  a different  TCP.  In  this  case,  when  the  retransmission  process 
awakened,  it  realized  that  a packet  had  to  be  retransmitted  ^ an  ACK  had 
to  be  sent.  Thus  at  least  one  retransmission  always  went  with  the  ACK.  As 
the  retransmission  timeout  decreased,  the  situation  accentuated  itself 
noticeably. 

To  more  precisely  observe  this  behavior,  we  separated  the  packet  retrans- 
mission timeout  from  the  nominal  retransmission  process  timeout.  Wo  also 
modified  the  TCP  code  to  allow  for  one  of  three  kinds  of  acknowledgment 
procedu'-e,  as  shown  below: 

(a)  send  an  ACK  immediately  upon  delivering  a packet  into  a user  buffer. 
Set  ‘'MEWRECnvr."  flag  whenever  an  acceptable  packet  (even  a duplicate) 
arrives  and  reset  this  flag  whenever  an  ACK  is  sent. 

(b)  same  as  (a)  for  NEWRECEIVE,  except  set  this  flag  on  delivering  data 
into  a user  buffer  rather  than  forcing  an  ACK  to  l>e  sent. 
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(c)  same  as  for  (a)  as  far  as  NfWRiCEIVE  flag  treatment,  but  send  an 
ACK  on  delivery  of  data  to  user  buffer  only  when  the  send  letter 
queue  is  empty,  otherwise,  set  the  NLWP.tCtlVE  flag  instead. 

We  have  tested  al tet'nati ves  (a)  and  (b)  with  a single  message  at  a 
time  transmission  regime,  witii  the  results  shown  in  table  4.  The  results  are 
not  entirely  consistent.  When  ACKs  are  not  forced,  and  both  timeouts  are 
0.5  seconos,  we  see  that  roughly  120(127  actually)  letters  and  retransmissions 
were  sent.  This  corresponds  to  one  letter  every  0.5  seconds.  Letters  are 
sent  roughly  as  often  as  ACKs  are  generated  every  1/2  second  by  the  retransmission 
process.  When  ACKs  are  sent  immediately  upon  delivery  of  new  data  (line  S,  Table  ^ 
the  number  of  retransmissions  drops  to  8 and  the  number  of  letters  jumps  to 
151.  The  strategy  of  waiting  for  the  retransmission  process  to  send  ACKs  (in 
the  absence  of  reverse  traffic)  is  clearly  a poor  one.  As  the  retransmission 
process  is  awakened  more  frequently,  to  reduce  the  ACK  delay,  more  and 
more  bandwidth  is  used  up  with  retransmissions  (lines  1-4). 

We  can  try  to  sketch  the  flow  of  events  which  account  for  the  behavior 
we  have  observed.  In  figure  7,  we  show  time  advancing  from  the  top  of  the 
figure  towards  the  bottom.  The  left  time  line  is  for  events  occurring  on 
the  send  side  of  the  TCP  while  the  right  time  line  is  for  events  on  the  receive 
side.  Transmission  of  information  back  and  forth  is  indicated  by  arrows 
sloping  downward  to  show  time  delay.  Taking,  for  example,  line  5 of  Table  4,  we  S( 
that  no  ACKs  were  ever  sent  by  thet  etransrnission  process  and  that  only  a 
few  retransmissions  occurred.  We  speculate  that  this  is  so  because,  most 
of  the  time,  a letter  (packet)  is  received,  delivered  to  a user  buffer,  and 
and  ACK  forced  out  ^ rec^-ived  (thereby  removing  the  packet  from  the  retrans- 
mission queue)  before  the  0.5  second  retransmission  time  expires.  I 
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I ft  fiqur('  o \v-‘  bliow  how  rotrariMii  i si,  ion  may  occur  for  tho  lino  5 
{ldbl(‘  0 case’.  Bdsirally,  tho  ACK  is  soriiofiow  dolayed  in  dol  ivory  so  tliat 
tho  t'ct  ransnri ss  ion  process  is  able  to  sciiodulo  a retransmission.  This  is  an 
alrtrniinq  r'osult  because  it  implies  tiiat  the  round  trip  time  for  the  message 
and  its  associated  ACf  can  exceed  0.5  seconds.  We  will  investigate  this 
phenomenon  (it  is  possible  tlial  tire  self-looping  and  interference  between 
send/receive  setsapttor inej  on  the  same  connection  is  the  source  of  this  odd 
behavior') . 

In  -.igure  9,  we  illustrviie  l.ow  ACKs  only  or  ACKs  and  retransmissions 
might  occur,  accourrting  for'  tire  statistics  in  line  1 of  table  4. 

It  can  be  seen  fr'oin  line  ? ef  Table  4 that  the  number  of  ACKs-only 
increase'  to  117  when  tr.e  ret rctosmission  process  is  run  twice  as  often.  This 
simply  is  the  result  of  runirm^  the  retransn.ission  process  before  the  packet 
has  been  delivered  to  tfie  u./ Line  .3  of  fable  4 shows  increases  in 
ACKs  only  a_nd  ACKs  with  retranv  ;issions  because  the  retransmission  process 
appeors  to  be  catcliiiig  the  hnmir.CLlvr  flag  both  when  set  on  packet  arrival 
^1^  when  set  because  data  has  been  deliverei  to  the  user. 

It  is  apparent  that  a basic  prob'eni  is  .ii’t  there  is  a long  delay  from 
the  time  a packet  arrives  until  the  tin. a it  is  delivered  to  the  user.  When 
the  retransmission  tiisicont  for  a packet  is  1/4  second,  there  is  a substantial 
increase  in  the  nuinber  of  retransmissions . We  will  investigate  this  further 
and  specifically  measure  tlie  delays  to  find  out  where  they  are  coming  from. 
One  conjecture  is  that  tfie  mul ti-process  implementation  uses  substantial 
overhead  in  the  LLF  sclieduler.  This  scliedulcr  is  run  after  most  system  calls 
(e.g.  Signal,  wait,  P,  V...)  and  could  be  using  a large  fraction  of  tlie  CPU 
cycles.  We  can  measure  this  and  will  report  on  it. 


7,2.7.  Multiple  letter  Ifirouqhput 

To  doteriiiine  wfiat  effect  "filling  the  pip<^"  migfit  have,  we  tried 
having  two  letters  "outstaridifig"  with  the  results  as  shown  in  Table  5. 

With  two  letters  outstanding  there  was  an  increase  in  throughput  and, 
with  9.5  second  retransmission  time,  tlie  ratio  of  letters/retransmissions 
remained  about  3:1.  The  nuniber  of  empty  ACKs  sent  dropped  substantial  ,y 
since  the  secor^d  letter  often  carried  the  ACK  for  the  first.  However, 
disaster  struck  when  tlie  retransmission  time  was  reduced  to  0.25  seconds 
when  2/3  of  a11  data  transmissions  were  retransmissions  and  letters  equalled 
ACKs  in  nuniber. 

The  statistics  (if  Table  5 come  from  an  acknowledgment  procedure  which 
delays  ACKs  until  the  retransmission  process  times  out  or  new  data  is  sent. 
We  expect  better  results  when  ACKs  are  forced  out  when  data  is  delivered  to 
the  user. 

3.0  Security  Work 

In  a separate  report  on  secure  packet  fragmentation,  we  discussed  a 
method  for  allowing  encrypted  packets  to  be  fragmented  at  an  internet 
gateway,  but  decrypted  without  reassembly. 
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TCPNET-VDH  interface 
TCPELF-EFL  calls 
subtotal 

TCP  user  calls  (TCPUSR) 

Send  Process  (TCPSND) 

Receive  Process  (TCPRCV) 

Input  Packet  Handler  (TCPINPKT) 
Retransmission  Process  (TCPRTN) 
subtotal  “ TCP  system 
FLEA  (debugging  package) 

EXERCISER  (traffic  generator/measurement) 
TOTAL 


1374 

764 

971 

1039 

4148 

1567 

1564 

947 

3323 

968 


7,769 

1,141 


12,517 

2,029 

1,659 

25,115 


TCP/ELF  Space  Requirement 
Table  1 
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System  Component 

Size  in 
16  bit  Words 

TCP  input  CO -routine 

600 

TCP  output  co-routine 

500 

with  retransmission 

Shared  ring  buffer 
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Mi  sc.  error  handling 
TCP-0  subtotal 

GOO 
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Interrupt  dispatch  (est.) 
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TELNET  (est.) 

1,500 

Terminal  I/O  (est. ) 

300 

HOST/PRNET  software 

? 

PR  Terminal  Software  total 

4,100+  ? 

TCP-0  Software  Sizes 
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No.  \CKs  Sent 

Test  Letter  Length  Retransmission  No.  Letters  by  Ketransmission  No. 

Durat ion  (octets)  Timeout  Sent  Process  Ret ransmissions 
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